Methods for performing fast detail-preserving image filtering

ABSTRACT

A method for performing fast detail-preserving filtering of an input digital image includes, for each pixel in the image, calculating the difference between the selected pixel and each of its four neighboring pixels located above, left, right, and below the selected pixel, calculating a scaled weighted sum of differences between the actual pixel and its four neighbors, where for each neighboring pixel the weight is an edge-sensing function having a data-adaptive scaling parameter function, and adding the weighted sum of differences to the value of the selected pixel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/446,274, filed Feb. 24, 2011.

BACKGROUND

1. Field of the Invention

The present invention relates to image processing. More particularly,the present invention relates to image filtering and specifically tofast detail-preserving image filtering.

2. The Prior Art

A digital image with R rows and S columns can be seen as atwo-dimensional array of pixels x_((r,s)), where r=1, 2, . . . , R ands=1, 2, . . . , S denote the row and column coordinates, respectively.The input image can be processed by the traditional anisotropicdiffusion filter as follows:

$\begin{matrix}{x_{({r,s})}^{\lbrack{i + 1}\rbrack} = {x_{({r,s})}^{\lbrack i\rbrack} + {\lambda {\sum\limits_{{({g,h})} \in \zeta}\; \left\lbrack {{f\left( {x_{({r,s})}^{\lbrack i\rbrack},x_{({g,h})}^{\lbrack i\rbrack}} \right)} \cdot \left( {x_{({r,s})}^{\lbrack i\rbrack} - x_{({g,h})}^{\lbrack i\rbrack}} \right)} \right\rbrack}}}} & (1)\end{matrix}$

where i denotes the iteration index, with i=0 indicating the input(unprocessed) image. The term ζ={(r−1,s), (r,s−1), (r,s+1), (r+1,s)}denotes the neighboring pixels located above, left, right, and belowwith respect to the pixel location (r,s) under consideration. The termƒ(·) denotes the edge-sensing function which can be defined as follows:

$\begin{matrix}{{f\left( {a,b} \right)} = \frac{1}{1 + {\left( {a - b} \right)^{2}/\kappa^{2}}}} & (2)\end{matrix}$

where κ is a predetermined parameter, for example, set as κ=2.

The filtering procedure first selects the actual pixel x_((r,s)) to befiltered. Then, the four neighbors for x_((g,h)), for (g,h)εζ andζ={(r−1,s),(r,s−1), (r,s+1), (r+1,s)}, located above, left, right, andbelow with respect to the pixel location (r,s) under consideration areselected. Note that the pixel location above, left to, right to andbelow the actual location (r,s) are indicated as (r−1,s), (r,s−1),(r,s+1), and (r+1,s), respectively. Then, the difference betweenx_((r,s)) and each of its four neighbors is calculated; these fourdifferences are indicated as x_((r,s)) ^([i])−x_((g,h)) ^([i]), for(g,h) εζ. Each of these four differences is associated with the weightf(x_((r,s)) ^([i]), x_((g,h)) ^([i])) calculated using Equation (2).Once the differences and weights are obtained, the actual pixelx_((r,s)) is adjusted by adding to its value the value ofλ·Σ[f(x_((r,s)) ^([i]), x_((g,h)) ^([i]))·(x_((r,s)) ^([i])−x_((g,h))^([i]))] which represents the weighted sum of differences between theactual pixel and its four neighbors, the sum is scaled by the parameter,λ which controls the level of this adjustment.

The filtering process then moves to next pixel location and executes theabove procedure. The procedure is repeated in all pixel locations of theinput image until all pixels are processed. This gives one iteration,which is indicated in Equation (1) using index [i+1] where [i] refers tothe input image (for i=0) or the image obtained in previous iteration(for i>0).

Anisotropic diffusion is an iterative filtering process; reasonablequality of the filtered image is usually achieved after a number ofiterations, which may not be suitable for real-time processing. Thisiterative processing requires performing Equation (1) in each pixellocation of the input image to produce an image to be used as the inputin the subsequent iteration.

BRIEF DESCRIPTION

A method for performing fast detail-preserving filtering of an inputdigital image includes, for each pixel in the image, calculating thedifference between the selected pixel a and each of its four neighboringpixels located above, left, right, and below the selected pixel,calculating a scaled weighted sum of differences between the actualpixel and its four neighbors, where for each neighboring pixel b theweight is an edge-sensing function having a data-adaptive scalingparameter function, and adding the weighted sum of differences to thevalue of the selected pixel a.

According to one aspect of the present invention, the weight may be

${{f\left( {a,b} \right)} = \frac{1}{1 + {\left( {a - b} \right)^{2}/{\kappa \left( {a,b} \right)}^{2}}}};$

and κ(·) is the data-adaptive scaling function.

According to another aspect of the invention, KO may be a function usingthe pixel under consideration to estimate noise variance. In oneillustrative embodiment, κ(a,b)=θ(Θ₀+Θ₁a+Θ₂a²) where Θ₀, Θ₁, and Θ₂ arepredetermined parameters.

According to a further aspect of the invention, κ(·) may be a functionusing the neighboring pixels to estimate noise variance. In oneillustrative embodiment, κ(a,b)=θ(Θ₀+Θ₁b+Θ₂b²).

According to a yet another aspect of the invention, κ(·) may be afunction using a combination of the pixel under consideration and theneighboring pixels to estimate noise variance. In one illustrativeembodiment, κ(a,b)=θ(Θ₀+Θ₁c+Θ₂c²), wherein the termc=(w_(a)a+w_(b)b)/(w_(a)+w_(b)) and wherein w_(a) and w_(b) arepredetermined weights associated with pixels a and b, respectively.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

FIG. 1 is a flow diagram showing an illustrative method for performingfast detail-preserving filtering of an input digital image according toone aspect of the present invention.

FIG. 2 is a flow diagram showing an illustrative method for performingfast detail-preserving filtering of an input digital image according toanother aspect of the present invention.

FIG. 3 is a flow diagram showing an illustrative method for performingfast detail-preserving filtering of an input digital image according toanother aspect of the present invention.

FIG. 4 is a flow diagram showing an illustrative method for performingfast detail-preserving filtering of an input color digital imageaccording to another aspect of the present invention.

DETAILED DESCRIPTION

Persons of ordinary skill in the art will realize that the followingdescription of the present invention is illustrative only and not in anyway limiting. Other embodiments of the invention will readily suggestthemselves to such skilled persons.

To speed-up the filtering process while achieving similar noisesmoothing performance as the above prior solution, the proposed solutionuses the edge-sensing function ƒ(·) with a data-adaptive scalingfunction κ(·), as follows:

$\begin{matrix}{{f\left( {a,b} \right)} = \frac{1}{1 + {\left( {a - b} \right)^{2}/{\kappa \left( {a,b} \right)}^{2}}}} & (3)\end{matrix}$

Following the notation used in Equation (1), the terms a and b denotethe actual pixel and its neighbor, respectively.

Since anisotropic diffusion is typically used in denoising applications,it is reasonable that κ(·) reflects characteristics of noise present inthe image. One possible solution is to estimate the noise level from theimage and then reflect this estimate in the design of κ(·). However,this approach may be too slow and complex, and may be accompanied withan estimation error that can lead to insufficient noise suppression whenthe noise level is underestimated or excessive smoothing when the noiselevel is overestimated. Therefore, a more practical solution, at leastwhen it comes to imaging devices and systems with known noisecharacteristics, is to evaluate image noise during sensor calibrationand them characterize this noise using the corresponding mathematicalmodel.

For example, in practice, noise variance can be characterized using thequadratic model with parameters Θ₀, Θ₁, and Θ₂ as follows:

κ(a,b)=θ(Θ₀+Θ₁ a+Θ ₂ a ²)  (4)

where θ is a scaling factor that can be used to control the amount ofdenoising. Namely, stronger smoothing effects can be obtained byincreasing the value of θ whereas better preservation of image detailscan be obtained by decreasing the value of θ. In general, parameter θcan be predetermined, or controlled by the user, or even calculatedadaptively using the statistics of the image under consideration.Example values of quadratic model parameters obtained in calibration canbe set as Θ₀=2.140387, Θ₁=0.027064, and Θ₂=0.000017. Scaling parameter θcan usually range from 0.1 to 10. Specific values of these designparameters may vary depending on the image sensor characteristics, imagequality requirements, application constraints, etc.

Referring now to FIG. 1, a flow diagram shows an illustrative method forperforming fast detail-preserving filtering of an input digital imageaccording to one aspect of the present invention wherein the scalingfactor is a function of the selected pixel. The process begins atreference numeral 10. At reference numeral 12, a pixel in the image isselected for processing. At reference numeral 14, the differencesbetween the selected pixel and its neighbors above, below, to the right,and to the left are calculated.

At reference numeral 16, a weighted and scaled sum of the differencesbetween the selected pixel and its four neighbors is calculated, withthe scaling function κ(·) being a function of the selected pixel. Atreference numeral 18, the weighted and scaled sum of the differences isadded to the selected pixel.

At reference numeral 20, it is determined whether the selected pixel wasthe last unprocessed pixel in the digital image. If so, the process endsat reference numeral 22. If the selected pixel was the not lastunprocessed pixel in the digital image, the process returns to referencenumeral 12, where another pixel in the image is selected for processing.If more than one iteration is to be performed, then the filtered imageproduced at reference numeral 22 is used in the subsequent iteration asthe input image at reference numeral 10 and the whole filteringprocedure (reference numerals 12 to 20) is repeated again.

Equation (4) shows an example of the scaling function κ(·) that uses thepixel under consideration to estimate the noise variance. Since thissample is shared for all (g,h) εζ, Equation (4) needs to be calculatedonly once per pixel location (r,s) denoised by Equation (1). Alternativesolutions can use the neighboring pixels instead, resulting in

κ(a,b)=θ(Θ₀+Θ₁ b+Θ ₂ b ²)  (5)

or the combination of the actual pixel a and its neighbor b as follows:

Θ(a,b)=θ(Θ₀+Θ₁ c+Θ ₂ c ²), with c=(w _(a) a+w _(b) b)/(w _(a) +w_(b))  (6)

where w_(a) and w_(b) are the weights associated with pixels a and b,respectively. Note that this invention is not limited to the aboveexamples, as it can work with some other forms of the scaling functionκ(·).

Referring now to FIG. 2, a flow diagram shows an illustrative method forperforming fast detail-preserving filtering of an input digital imageaccording to another aspect of the present invention wherein the scalingfactor is a function of the neighboring pixels. The process begins atreference numeral 30. At reference numeral 32, a pixel in the image isselected for processing. At reference numeral 34, the differencesbetween the selected pixel and its neighbors above, below, to the right,and to the left are calculated.

At reference numeral 36, a weighted and scaled sum of the differencesbetween the selected pixel and its four neighbors is calculated, withthe scaling function κ(·) being a function of the neighboring pixels. Atreference numeral 38, the weighted and scaled sum of the differences isadded to the selected pixel.

At reference numeral 40, it is determined whether the selected pixel wasthe last unprocessed pixel in the digital image. If so, the process endsat reference numeral 42. If the selected pixel was the not lastunprocessed pixel in the digital image, the process returns to referencenumeral 32, where another pixel in the image is selected for processing.If more than one iteration is to be performed, then the filtered imageproduced at reference numeral 42 is used in the subsequent iteration asthe input image at reference numeral 30 and the whole filteringprocedure (reference numerals 32 to 40) is repeated again.

Referring now to FIG. 3, a flow diagram shows an illustrative method forperforming fast detail-preserving filtering of an input digital imageaccording to another aspect of the present invention wherein the scalingfactor is a function of the selected pixel and its neighbors. Theprocess begins at reference numeral 50. At reference numeral 52, a pixelin the image is selected for processing. At reference numeral 54, thedifferences between the selected pixel and its neighbors above, below,to the right, and to the left are calculated.

At reference numeral 56, a weighted and scaled sum of the differencesbetween the selected pixel and its four neighboring pixels iscalculated, with the scaling function κ(·) being a function of theselected pixel and its neighbors. At reference numeral 58, the weightedand scaled sum of the differences is added to the selected pixel.

At reference numeral 60, it is determined whether the selected pixel wasthe last unprocessed pixel in the digital image. If so, the process endsat reference numeral 62. If the selected pixel was the not lastunprocessed pixel in the digital image, the process returns to referencenumeral 52, where another pixel in the image is selected for processing.If more than one iteration is to be performed, then the filtered imageproduced at reference numeral 62 is used in the subsequent iteration asthe input image at reference numeral 50 and the whole filteringprocedure (reference numerals 52 to 60) is repeated again.

In case of color images, a straightforward extension of the method ofthe present invention is to apply the method of the present invention toeach color channel separately. Advanced solutions, with the focus onpreserving the color correlation, should use a vector form of Equation(3). In one example, (a−b)²/κ(a,b)² in Equation (3) can be replaced bythe sum or average value of (a−b)²/κ(a,b)² values obtained for eachcolor channel. In another example, (a−b)² can be replaced with adistance or similarity measure, such as the Euclidean distance or theMinkowski metric, for quantifying the difference between two colorvectors. The value obtained using this distance measure should be scaledby the sum or average value of κ(a,b)² obtained for each color channel.These advanced solutions ensure consistent processing across colorchannels due to using consistent weights for each color component at thepixel location being filtered.

Referring now to FIG. 4, a flow diagram shows an illustrative method forperforming fast detail-preserving filtering of an input digital imageaccording to one aspect of the present invention wherein the scalingfactor is a function of the selected pixel. The process begins atreference numeral 70. At reference numeral 72, a pixel in the image isselected for processing. At reference numeral 74, the differencesbetween the selected pixel and its neighbors above, below, to the right,and to the left are calculated.

At reference numeral 76, a weighted and scaled sum of the differencesbetween the selected pixel and its four neighbors is calculated, withthe scaling function κ(·) being a function of the selected pixel. Atreference numeral 78, the weighted and scaled sum of the differences isadded to the selected pixel.

At reference numeral 80, it is determined whether the selected pixel wasthe last unprocessed pixel in the digital image. If so, the process endsat reference numeral 82. If the selected pixel was the not lastunprocessed pixel in the digital image, the process returns to referencenumeral 72, where another pixel in the image is selected for processing.If more than one iteration is to be performed, then the filtered imageproduced at reference numeral 82 is used in the subsequent iteration asthe input image at reference numeral 70 and the whole filteringprocedure (reference numerals 72 to 80) is repeated again.

While embodiments and applications of this invention have been shown anddescribed, it would be apparent to those skilled in the art that manymore modifications than mentioned above are possible without departingfrom the inventive concepts herein. The invention, therefore, is not tobe restricted except in the spirit of the appended claims.

1. A method for creating an output digital image by performing fastdetail-preserving filtering of an input digital image comprising: a)selecting a pixel a from the input digital image, pixel a not being in arow or column at the boundary of the image; b) calculating thedifference between the selected pixel and each of its four neighboringpixels located above, left, right, and below the selected pixel; c)calculating a scaled weighted sum of differences between the selectedpixel and its four neighbors, where for each neighboring pixel b theweight is an edge-sensing function having a data-adaptive scalingparameter function; d) adding the weighted sum of differences to thevalue of the selected pixel; and performing a) through d) for each pixelin the input digital image that is not in a row or column at theboundary of the image.
 2. The method of claim 1 wherein: the weight is${{f\left( {a,b} \right)} = \frac{1}{1 + {\left( {a - b} \right)^{2}/{\kappa \left( {a,b} \right)}^{2}}}};$and κ(·) is the data-adaptive scaling function
 3. The method of claim 2wherein κ(·) is a function using the pixel under consideration toestimate noise variance.
 4. The method of claim 3 whereinκ(a,b)=θ(Θ₀+Θ₁a+Θ₂a²).
 5. The method of claim 2 wherein κ(·) is afunction using the neighboring pixels to estimate noise variance.
 6. Themethod of claim 5 wherein κ(a,b)=θ(Θ₀+Θ₁b+Θ₂b²).
 7. The method of claim2 wherein κ(·) is a function using a combination of the pixel underconsideration and the neighboring pixels to estimate noise variance. 8.The method of claim 7 wherein κ(a,b)=θ(Θ₀+Θ₁c+Θ₂c²), wherein the termc=(w_(a)a+w_(b)b)/(w_(a)+w_(b)) and wherein w_(a) and w_(b) are weightsassociated with pixels a and b, respectively.
 9. A method for creatingan output color digital image by performing fast detail-preservingfiltering of an input color digital image comprising: a) selecting apixel a from the input color digital image, pixel a not being in a rowor column at the boundary of the image; b) calculating separately foreach color channel in the selected pixel the difference between theselected pixel and each of its four neighboring pixels located above,left, right, and below the selected pixel; c) calculating separately foreach color channel in the selected pixel a scaled weighted sum ofdifferences between the selected pixel and its four neighbors, where foreach neighboring pixel b the weight is an edge-sensing function having adata-adaptive scaling parameter function; d) adding separately for eachcolor channel in the selected pixel the weighted sum of differences tothe value of the selected pixel; and performing a) through d) for eachpixel in the input color digital image that is not in a row or column atthe boundary of the image.
 10. The method of claim 9 wherein: the weightis${{f\left( {a,b} \right)} = \frac{1}{1 + {\left( {a - b} \right)^{2}/{\kappa \left( {a,b} \right)}^{2}}}};$and κ(·) is the data-adaptive scaling function
 11. The method of claim10 wherein κ(·) is a function using the pixel under consideration toestimate noise variance.
 12. The method of claim 11 whereinκ(a,b)=θ(Θ₀+Θ₁a+Θ₂a²).
 13. The method of claim 10 wherein κ(·) is afunction using the neighboring pixels to estimate noise variance. 14.The method of claim 13 wherein κ(a,b)=θ(Θ₀+Θ₁b+Θ₂b²).
 15. The method ofclaim 10 wherein κ(·) is a function using a combination of the pixelunder consideration and the neighboring pixels to estimate noisevariance.
 16. The method of claim 15 wherein κ(a,b)=θ(Θ₀+Θ₁c+Θ₂c²),wherein the term c=(w_(a)a+w_(b)b)/(w_(a)+w_(b)) and wherein w_(a) andw_(b) are weights associated with pixels a and b, respectively.
 17. Themethod of claim 9 wherein: the weight${{f\left( {a,b} \right)} = \frac{1}{1 + \eta}};$ η is a function of(a−b)²/κ(a,b)² values obtained for each color channel; and κ(·) is thedata-adaptive scaling function.
 18. The method of claim 9 wherein: theweight ${{f\left( {a,b} \right)} = \frac{1}{1 + {\eta_{1}/\eta_{2}}}};$η₁ is a combination of a−b values obtained for each color; η₂ is acombination of κ(a,b)² values obtained for each color; and κ(·) is thedata-adaptive scaling function.
 19. The method of claim 9 wherein: theweight is a combination of${f\left( {a,b} \right)} = \frac{1}{1 + {\left( {a - b} \right)^{2}/{\kappa \left( {a,b} \right)}^{2}}}$values obtained for each color channel; and κ(·) is the data-adaptivescaling function